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ABSTRACT  |  For  future  systems  that  require  one  or  a  small 
team  of  operators  to  supervise  a  network  of  automated  agents, 
automated  planners  are  critical  since  they  are  faster  than  hu¬ 
mans  for  path  planning  and  resource  allocation  in  multivariate, 
dynamic,  time-pressured  environments.  However,  such  plan¬ 
ners  can  be  brittle  and  unable  to  respond  to  emergent  events. 
Human  operators  can  aid  such  systems  by  bringing  their 
knowledge-based  reasoning  and  experience  to  bear.  Given  a 
decentralized  task  planner  and  a  goal-based  operator  interface 
for  a  network  of  unmanned  vehicles  in  a  search,  track,  and 
neutralize  mission,  we  demonstrate  with  a  human-on-the-loop 
experiment  that  humans  guiding  these  decentralized  planners 
improved  system  performance  by  up  to  50%.  However,  those 
tasks  that  required  precise  and  rapid  calculations  were  not 
significantly  improved  with  human  aid.  Thus,  there  is  a  shared 
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space  in  such  complex  missions  for  human-automation 
collaboration. 

KEYWORDS  |  Command  and  control;  decentralized  task  plan¬ 
ning;  decision  support  systems;  human-automation  interac¬ 
tion;  human  supervisory  control;  unmanned  vehicles 

I.  INTRODUCTION 

The  use  of  unmanned  vehicles  (UVs)  has  recently  revolu¬ 
tionized  U.S.  military  operations.  In  the  past  year,  the  U.S. 
Air  Force  reached  an  important  milestone  in  that  it  now 
has  more  unmanned  aerial  vehicles  (UAVs)  than  manned 
aircraft.  Other  countries  are  following  this  lead,  with  re¬ 
cent  significant  advances  in  UAV  heavy-lift  capacity  by 
Israel  and  the  United  Kingdom.  These  dramatic  advances 
are  not  only  limited  to  the  military  as  the  international 
civil  sector  also  looks  to  such  unmanned  technologies  to 
aid  operations  such  as  fighting  forest  fires,  undersea  ex¬ 
ploration,  monitoring  wildlife,  inspecting  bridges,  and 
supporting  first  responders  such  as  police  and  rescue 
organizations.  UAV  expenditures  alone  are  predicted  to 
more  than  double  in  the  next  ten  years,  and  are  expected  to 
exceed  $80  billion  [1]. 

Accompanying  these  impressive  advances  in  UV  plat¬ 
forms  are  equally  dramatic  leaps  in  intelligent  UV  control 
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systems.  Point-and-click  UAV  operations  are  routine  such 
that  operators  only  have  to  click  a  place  on  a  map  to  com¬ 
mand  a  vehicle,  and  the  onboard  autonomy  determines  the 
most  appropriate  control  actions.  Such  advances  over 
legacy  systems  that  require  typical  stick-and-rudder  skills 
have  revolutionized  military  operations  in  that  traditional 
pilots  are  no  longer  needed  to  control  such  systems. 
Indeed,  while  not  yet  operational,  research  and  develop¬ 
ment  across  a  number  of  laboratories  has  shown  that 
networks  of  UAVs,  unmanned  ground  vehicles,  and  even 
other  UV  types  such  as  unmanned  underwater  vehicles 
and  unmanned  surface  vessels  can  be  controlled  by  a 
single  operator  [2]  — [5] .  This  single-operator,  multiple- 
unmanned-vehicle  architecture  is  the  crux  of  the  Depart¬ 
ment  of  Defense’s  vision  for  network-centric  operations, 
where  a  network  of  intelligent  UVs  is  able  to  semiauto- 
nomously  work  with  a  small  group  of  human  operators  to 
execute  dynamic  missions  in  time-critical  scenarios  [6]. 

While  all  of  the  previously  mentioned  studies  exam¬ 
ined  the  ability  of  underlying  automation  (in  the  form  of 
planning  and  control  algorithms)  to  control  a  network  of 
heterogeneous  unmanned  vehicles  (UxVs),  a  significant 
limitation  of  this  work  is  a  lack  of  investigation  of  critical 
human-automation  collaboration  issues.  In  all  future  vi¬ 
sions  of  operating  networks  of  UVs,  humans  are  expected 
to  assume  some  kind  of  supervisory  role,  through  high- 
level  goal  expressions  which  could  include  resource  alloca¬ 
tion,  target  designation,  approval  for  weapons  release,  etc. 
While  research  has  shown  that  one  operator  can  theoret¬ 
ically  control  multiple  UVs  with  varying  levels  of  em¬ 
bedded  autonomy  [7] -[10],  there  have  been  no  previous 
studies  that  examine  how  an  operator  interacts  with  de¬ 
centralized  UV  planners.  Moreover,  whether  such  human 
interaction  can  improve,  or  possibly  degrade,  overall  sys¬ 
tem  performance  has  not  previously  been  addressed.  This 
paper  fills  this  gap  by  detailing  an  experiment  conducted  to 
investigate  what  impact  a  human  operator  has  on  the 
overall  performance  of  a  decentralized  UV  network,  and 
how  such  interactions  could  be  improved  to  enhance 
overall  human-system  performance. 

II.  DECENTRALIZED  PLANNING  FOR 
MULTIPLE  UV  MANAGEMENT 

We  developed  the  Onboard  Planning  System  for  UxVs 
Supporting  Expeditionary  Reconnaissance  and  Surveil¬ 
lance  (OPS-USERS)  system  to  provide  a  planning  frame¬ 
work  for  a  team  of  autonomous  agents  under  human 
supervisory  control  participating  in  expeditionary  mis¬ 
sions,  which  rely  heavily  on  intelligence,  surveillance,  and 
reconnaissance.  The  mission  environment  contains  an 
unknown  number  of  mobile  targets,  each  of  which  may  be 
friendly,  hostile,  or  unknown.  The  mission  scenario  is 
multiobjective,  and  includes  finding  as  many  targets  as 
possible,  keeping  accurate  position  estimates  of  hostile  and 
unknown  targets,  and  neutralizing  all  hostile  targets.  It  is 


assumed  that  static  features  in  the  environment,  such  as 
terrain  type,  are  known  a  priori,  but  dynamic  features,  such 
as  target  locations,  are  not. 

A.  Architecture 

The  OPS-USERS  system  architecture  is  specifically  de¬ 
signed  to  meet  the  challenges  associated  with  an  auto¬ 
mated  decision-making  system  integrated  with  a  human 
operator  on  the  loop.  Two  key  challenges  are:  1)  balancing 
the  roles  and  responsibilities  of  the  human  operator  and 
the  automated  planner,  and  2)  optimizing  resource  alloca¬ 
tion  to  accomplish  each  of  the  mission  objectives.  The 
system  relies  on  the  relative  strengths  of  both  humans  and 
automation  in  that  a  human  operator  provides  valuable 
judgment  and  field  experience  in  recognizing  patterns  and 
defining  new  goals  as  environmental  changes  dictate.  A 
significant  benefit  of  automation  is  its  ability  to  provide 
raw  computational  power  and  rapid  optimization  capabil¬ 
ity  on  the  order  of  seconds  for  resource  allocation  prob¬ 
lems  that  take  human  decision  makers  minutes  or  hours  to 
solve. 

In  OPS-USERS,  decision  making  responsibility  is  la¬ 
yered  to  promote  goal-based  reasoning  such  that  the  hu¬ 
man  guides  the  autonomy,  but  the  automation  assumes  the 
bulk  of  the  computation.  The  automated  planner  is  respon¬ 
sible  for  decisions  requiring  rapid  calculations  or  optimi¬ 
zation,  and  the  human  operator  supervises  the  planner  for 
high-level  goals  such  as  where  to  focus  the  search  and 
which  tasks  should  be  included  in  the  overall  plan,  as  well 
as  those  tasks  that  require  strict  human  approval,  such  as 
weapons  release.  Fig.  1  depicts  the  role  allocation  balance 
between  the  human  and  automation. 

In  this  goal-based  system,  the  operator  is  responsible 
for  strategic  decision  making  that  includes  prioritizing 
which  tasks  should  be  performed  at  the  current  stage  in  the 
mission.  The  operator  interface  that  allows  such  interac¬ 
tions  is  discussed  in  detail  in  Section  II-F.  The  automated 
planner  is  responsible  for  tactical  planning  by  deciding 
which  UxVs  perform  which  task,  and  forming  a  task 
schedule  for  each  UxV.  Together,  the  human  operator  and 
the  automation  work  jointly  to  define  plans  that  meet 
operational  goals.  For  example,  the  automation  helps  the 


Fig.  1.  Decision  making  allocation  between  human  operator  and 
automated  planner. 
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human  by  estimating  when  and  where  the  UxVs  should 
revisit  targets  based  on  observed  target  behavior  and  sens¬ 
ing  capabilities  onboard  the  UxVs,  such  as  present  day 
motion  target  indicators.  The  human  helps  the  automation 
by  guiding  it  to  areas  the  operator  believes  are  more  likely 
to  contain  undetected  targets.  Thus,  this  overall  frame¬ 
work  allows  the  human  to  make  global  decisions  regarding 
locations  and  targets  of  interest,  while  the  automated 
planner  optimizes  the  local  trajectory  of  each  UxV  to 
maximize  searching  and  tracking. 

The  autonomy  only  manages  uncertainty  related  to  the 
targets,  which  is  modeled  by  spatial  probability  distribu¬ 
tions  and  visually  represented  to  operators  as  grayed  out 
regions  on  the  interface.  The  operator  aids  in  target  uncer¬ 
tainty  management  by  reasoning  about  where  unfound  or 
lost  targets  may  hide  and  create  search  tasks  to  guide  the 
UxVs  to  these  areas.  The  autonomy  maintains  these  distri¬ 
butions  over  time,  updating  them  as  the  UxVs  gather  new 
information  and  locally  optimizes  the  UxV  paths  (within  a 
limited  planning  horizon  due  to  computational  con¬ 
straints).  The  uncertainty  related  to  the  predicted  future 
location  of  the  targets  is  also  managed  by  the  autonomy 
through  periodic  revisits.  There  is,  however,  no  uncer¬ 
tainty  in  the  planning  itself,  i.e.,  the  system  does  not 
handle  uncertainty  in  the  UxVs’  states,  health,  or  capabi¬ 
lities,  nor  in  the  execution  of  tasks. 

B.  Mission  Tasks 

The  first  task  type  in  OPS-USERS  is  to  find  targets, 
called  the  search  task,  which  is  a  collaborative  effort  be¬ 
tween  the  human  and  the  automation.  In  OPS-USERS,  the 
UVs  are  equipped  with  a  variety  of  sensors  for  observing 
and  searching  the  environment,  and  a  target  may  be  de¬ 
tected  if  its  position  lies  within  an  UxV’s  sensor  footprint. 
The  automation  and  the  human  operator  work  in  a  com¬ 
plementary  manner  to  accomplish  efficient  searching.  This 
architecture  gives  the  system  flexibility  such  that  the  auto¬ 
nomy  makes  search  decisions  even  in  the  absence  of  ope¬ 
rator  input,  but  also  leverages  their  expertise  to  improve 
the  search  process,  as  discussed  in  Section  II-F. 

The  automation  generates  paths  for  the  UxVs  that 
maximize  the  likelihood  of  detecting  new  targets.  How¬ 
ever,  the  computational  complexity  associated  with  this 
optimization  process  grows  exponentially  with  the  length 
of  the  planned  paths  that  limits  the  automation  to  calcu¬ 
lating  localized  trajectories.  The  human  can  guide  the 
automation  and  prevent  myopic  behaviors  by  creating 
search  tasks  in  remote,  unsearched  areas  that  are  likely  to 
contain  new  targets.  The  automated  planner  then  assigns 
UxVs  to  the  search  areas  the  operator  has  identified  and 
optimizes  the  local  trajectories  of  these  UxVs  to  efficiently 
search  the  areas  of  interest. 

The  second  task  type  is  tracking  a  found  target,  called 
the  track  task,  which  is  a  task  supervised  by  the  human 
operator,  but  executed  by  the  automation.  Tracking  a 
target  for  a  period  of  time  decreases  the  uncertainty 


associated  with  that  target’s  state  (i.e.,  position  and  velo¬ 
city).  However,  targets  are  typically  not  tracked  continu¬ 
ously  in  order  to  allow  UxVs  the  flexibility  of  performing 
more  tasks.  When  a  track  task  ends,  the  tracking  UxV 
instantiates  a  new  track  task  so  that  the  target  will  be 
revisited  in  the  future.  The  revisit  time  is  calculated  based 
on  the  expected  growth  rate  of  the  target’s  position  uncer¬ 
tainty  and  the  UxVs’  sensing  capabilities. 

If  a  target  being  tracked  is  known  to  be  hostile,  a 
neutralize  hostile  task  is  also  created.  This  third  task  type 
involves  engaging  an  enemy  target,  and  can  only  be  per¬ 
formed  by  a  weaponized  UAV  (WUAV).  The  operator  must 
verify  the  classification  of  the  enemy  target  as  hostile  and 
approve  the  weapon  release. 

The  final  task  is  the  refuel  task,  which  requires  that 
each  UxV  return  to  base  to  refuel  before  its  fuel  supply  is 
depleted.  Refuel  times  are  also  scheduled  by  the  task 
planner  to  maximize  the  overall  mission  performance,  and 
are  treated  as  hard  constraints.  UxVs  may  decide  to  refuel 
early  if  that  enables  them  to  accomplish  more  tasks  on 
time.  This  health  management  task  is  handled  entirely  by 
the  automation  to  reduce  the  operator’s  workload. 

In  order  to  allow  the  human  and  the  automation  to 
collaborate  for  task  execution,  the  basic  system  architec¬ 
ture  is  divided  into  two  major  components,  as  shown  in 
Fig.  2.  The  first  is  the  distributed  tactical  planner,  which  is 
a  network  of  onboard  planning  modules  (OPMs)  [11]  that 
provides  coordinated  autonomy  between  the  UxVs.  Each 
UxV  carries  a  processor  that  runs  an  instance  of  the  OPM. 
The  second  is  the  ground  control  station,  which  consists  of 
a  centralized  strategic  planner  called  the  central  mission 
manager  (CMM),  and  the  operator  interface  (OI).  These 
two  components  are  discussed  in  turn. 

C.  The  Distributed  Tactical  Planner 

A  decentralized  implementation  was  chosen  for  the 
tactical  planner  to  allow  rapid  reaction  to  changes  in  the 
environment.  When  appropriate,  the  decentralized  task 
planner  may  modify  the  task  assignment  without  affecting 
the  overall  plan  quality  (e.g.,  UxVs  swap  tasks),  and  it  is 
able  to  make  these  local  repairs  faster  through  inter-UxV 
communication  than  if  it  had  to  wait  for  the  decision  to 
come  from  the  human  or  a  centralized  planner.  Further¬ 
more,  plans  can  be  carried  out  even  if  the  communication 
link  with  the  ground  control  station  is  intermittent  or  lost. 
The  architecture  is  scalable,  since  additional  UxVs  also  add 
computational  capability,  and  the  decentralized  frame¬ 
work  is  robust  to  a  single  point  of  failure,  since  no  single 
UxV  is  globally  planning  for  the  fleet. 

The  decentralized  task  planner  used  in  OPS-USERS  is 
the  consensus-based  bundle  algorithm  (CBBA),  a  decen¬ 
tralized,  polynomial-time,  market-based  protocol  [12]. 
CBBA  consists  of  two  phases  that  alternate  until  the  as¬ 
signment  converges.  In  the  first  phase,  task  selection,  UxVs 
select  the  set  of  tasks  for  which  they  receive  the  highest 
reward.  The  UxVs  place  bids  on  the  tasks  they  choose 
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Fig.  2.  OPS-USERS  system  architecture. 


where  the  bids  represent  the  marginal  improvement  in  the 
score  of  their  plan.  Thus,  task  planning  only  requires  the 
exchange  of  bids,  which  each  UxV  makes  based  on  its  own 
state  only  (i.e.,  the  UxVs  only  need  to  know  how  much  the 
other  UxVs  value  the  tasks  they  compete  for — the  bids — 
and  not  the  locations  or  trajectories  of  the  other  UxVs).  In 
the  second  phase,  conflict  resolution,  plan  information  is 
exchanged  between  neighbors  and  tasks  go  to  the  highest 
bidder.  CBBA  is  guaranteed  to  reach  a  conflict-free  assign¬ 
ment,  given  a  strongly  connected  network. 

One  key  advantage  of  CBBA  is  its  ability  to  solve  the 
multiple  assignment  problem  where  each  UxV  is  assigned  a 
set  of  tasks  (a  plan),  as  opposed  to  solving  the  single 
assignment  problem,  where  each  UxV  is  only  assigned  to 
their  next  task.  Planning  several  tasks  into  the  future  im¬ 
proves  effectiveness  in  complex  missions.  An  additional 
advantage  is  that  CBBA  runs  in  polynomial  time  with  re¬ 
spect  to  the  number  of  UxVs  and  the  number  of  tasks. 
Under  the  implementation  used  for  this  experiment, 
plans  could  be  generated  for  four  UxVs  and  ten  tasks  in 
less  than  1  s. 

A  successful  distributed  tactical  planner  must  consider 
both  task  assignment  as  well  as  path  planning.  In  OPS- 
USERS,  the  OPM  generates  paths  in  two  ways.  If  the  next 
task  in  an  UxV’s  plan  is  scheduled  such  that  the  UxV  must 
travel  directly  to  the  task  location,  the  UxV  is  said  to  have 
no  spare  time.  In  the  case  where  an  UxV  has  no  spare  time, 
it  calculates  the  minimum-time  path  to  the  task  location 
using  Dijkstra’s  algorithm  [13].  The  UxVs  avoid  both  static 
obstacles  and  dynamic  obstacles  (i.e.,  other  UxVs)  along 
the  way.  It  is  assumed  that  UxVs  are  aware  of  static  ob¬ 
stacles  a  priori,  and  they  have  an  estimate  of  dynamic  ob¬ 
stacles  based  on  communicated  trajectories  between  UxVs. 

In  the  case  where  the  next  task  in  an  UxV’s  plan  is 
scheduled  such  that  the  UxV  has  more  than  enough  time  to 
travel  to  the  task  location,  the  UxV  is  said  to  have  spare 


time.  In  this  case,  the  UxV  implements  a  spare  time  strat¬ 
egy,  which  is  chosen  by  the  operator  for  each  UxV  before 
the  mission  begins.  If  the  spare  time  strategy  is  to  loiter, 
the  UxV  generates  a  trajectory  to  stay  near  their  last  task 
location  until  they  run  out  of  spare  time  and  need  to  travel 
to  their  next  task  location.  If  the  UxV’s  spare  time  strategy 
is  to  search,  it  generates  a  search  trajectory  that  maximizes 
the  expected  probability  of  detecting  new  targets  along  the 
path.  The  UxV  builds  a  breadth-first,  depth-limited  search 
tree  over  a  receding  horizon  and  determines  which  cells 
are  observable  to  its  sensor  along  each  path.  The  UxV  se¬ 
lects  the  path  that  has  the  highest  cumulative  probability  of 
revealing  new  targets.  This  trajectory  generation  algorithm 
scales  exponentially  with  the  depth  of  the  search  (i.e., 
number  of  waypoints  in  the  trajectory)  and  the  size  of  the 
environment  (i.e.,  number  of  cells  in  the  world).  For  this 
reason,  the  planning  horizon  must  be  chosen  such  that  the 
trajectory  can  be  calculated  at  a  rate  at  least  as  fast  as  the 
planning  loop.  For  this  experiment,  a  planning  horizon  of 
four  waypoints  was  sufficient  for  the  planning  rate  of  1  Hz. 
The  search  effort  is  also  coordinated  across  the  fleet  to 
avoid  redundant  effort,  which  is  known  to  improve  search 
performance  [14]— [16] .  UxVs  communicate  their  trajecto¬ 
ries  to  each  other,  and  UxVs  receive  no  additional  reward 
for  observing  cells,  which  will  already  be  observable  along 
another  UxV’s  planned  path. 

D.  Maintaining  Situational  Awareness 

Each  autonomous  UxV  maintains  situational  awareness 
by  keeping  an  estimate  of  several  mission  parameters. 
Targets  that  are  yet  to  be  found  are  called  search  targets, 
and  there  are  an  unknown  number  of  such  targets  at  a 
given  time.  The  uncertainty  associated  with  the  location  of 
search  targets  is  represented  by  a  probability  map  [17]  — 
[19],  which  is  a  discretized  map  of  the  environment.  Each 
cell  i  in  the  map  has  some  probability  pf(t)  of  containing  at 
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least  one  search  target  at  time  t.  At  each  time  step,  the 
probabilities  are  propagated  according  to  a  target  motion 
model.  The  target  model  quantifies  the  expected  proba¬ 
bility  that  a  target  moves  from  one  cell  to  another  cell  in 
one  time  step.  This  transition  probability  is  given  by 

trans  __  Vtgt  ^ 

^  ftfree(0  eij 

The  target  model  assumes  that  each  target  is  none- 
vasive  in  that  its  motion  is  independent  of  the  actions 
taken  by  the  UxVs.  It  is  also  assumed  that  the  probability 
that  a  target  traveling  from  a  cell  i  to  a  cell  j  is  proportional 
to  vtgt,  the  target’s  expected  speed,  and  dt,  the  time  be¬ 
tween  updates.  It  is  assumed  inversely  proportional  to 
nfree(i),  the  number  of  nonobstructed  cells  adjacent  to  cell 
i,  and  to  ei;  ,  the  length  of  the  edge  of  cell  i,  in  the  direction 
of  cell  j.  vtgt  is  estimated  in  advance  of  a  mission  by  a 
mission  planner,  based  on  the  expected  target  types,  which 
can  include  heterogeneous  speeds.  When  a  cell  i  is  ob¬ 
served  by  an  UxV  at  time  t,  its  probability  of  containing  a 
new  target  is  updated  according  to 

Pi(t  +  !)  =Pi(0  •  (!-/•( 0) 

where  fi(t)  is  the  fraction  of  cell  i  that  was  observed  at 
time  t. 

When  a  new  target  is  found,  it  is  designated  a  track 
target.  Position,  velocity,  and  position  uncertainty  esti¬ 
mates  are  kept  for  each  track  target  in  the  environment.  If 
a  track  target  is  not  observed  by  any  UxV  at  time  t ,  the 
target’s  position  uncertainty  increases  by  a  factor  that  is 
proportional  to  the  target’s  last  known  velocity.  The 
growth  rate  of  the  position  uncertainty  of  a  given  track 
target  is  used  by  the  planner  to  schedule  the  next  desired 
revisit  time  for  that  target.  If  the  uncertainty  for  a  given 
target  becomes  too  large,  the  target  is  considered  lost,  and 
becomes  a  search  target. 

The  distributed  tactical  planner  runs  at  a  rate  of  one 
cycle  per  second.  Each  time  through  the  planning  loop, 
UxVs  share  information  regarding  their  current  position, 
their  current  planned  trajectory,  and  updates  on  target 
position  estimates.  UxVs  are  able  to  update  their  own 
situational  awareness  based  on  messages  from  other  UxVs. 

E.  The  Ground  Control  Station 

The  ground  control  station  is  remotely  connected  to 
the  UxV  network  and  primarily  consists  of  the  CMM  and 
the  OI.  The  CMM  is  a  centralized  computational  aid,  de¬ 
signed  to  assist  the  operator  in  strategic  decision  making. 
The  CMM  runs  a  local  centralized  version  of  the  task 
planner  to  determine  the  feasibility  of  assigning  a  given 
subset  of  tasks.  The  operator  is  able  to  modify  the  set  of 


tasks  under  consideration,  and  investigate  a  variety  of 
“what  if”  scenarios,  i.e.,  the  operator  changes  the  assigned 
tasks  and  then  evaluates  the  impact  of  the  new  schedule 
before  actually  sending  the  new  plan  to  the  UxV  team.  The 
operator  must  approve  every  automation-generated  plan 
before  the  tasks  are  released  to  the  distributed  tactical 
planner. 

This  strategic  level  planning  does  not  concern  the  ope¬ 
rator  with  the  mechanics  of  how  the  tasks  will  be  carried 
out  (i.e.,  which  UxV  will  perform  which  task),  but  allows 
the  operator  to  decide  which  tasks  should  be  included  in 
the  plan.  This  goal-based  approach,  as  compared  to  indi¬ 
vidual  vehicle  control,  is  critical  for  single-operator  control 
of  multiple  UVs  since  it  substantially  reduces  operator 
workload,  and  does  not  require  them  to  use  valuable  cog¬ 
nitive  skills  that  are  best  reserved  for  knowledge-based 
decisions  [7]. 

Quantitative  information  (i.e.,  UxV  positions  and  fuel 
levels,  target  position  estimates)  and  candidate  plans  are 
passed  from  the  CMM  to  the  OI,  which  converts  this  in¬ 
formation  into  a  visual  form  that  is  meaningful  to  the 
operator  (discussed  in  depth  in  the  next  section).  The 
operator  can  query  the  planner  to  determine  the  feasibility 
of  inserting  a  new  task  into  the  current  task  list.  Once  the 
operator  has  decided  on  a  plan,  the  approved  task  set  is 
sent  to  the  distributed  tactical  planner  via  the  CMM  for 
execution. 

F.  The  Operator  Interface 

In  order  to  allow  a  single  operator  the  ability  to  control 
multiple  heterogeneous  UVs  given  the  automated  planner 
previously  described,  two  interfaces  were  designed:  the 
map  display  and  the  schedule  comparison  tool,  detailed 
next. 

The  map  display  (Fig.  3)  represents  the  primary  OI, 
which  shows  both  geospatial  and  temporal  mission  infor¬ 
mation  (i.e.,  a  timeline  of  mission  events).  Icons  represent 
UxVs;  low,  medium,  and  high  priority  targets;  and  search 
tasks.  The  symbology  is  consistent  with  MIL-STD  2525 
[20].  This  interface  also  supports  an  instant  messaging 
communication  tool,  which  is  a  commonly  used  interface 
in  military  operations.  This  “chat”  tool  simulates  an  in¬ 
stant  messaging  tool  connected  to  a  command  center  that 
provides  high-level  direction  and  intelligence  about  targets 
in  a  designated  area.  In  this  experiment,  this  chat  tool  was 
used  to  communicate  permission  to  fire  orders  and  target 
priority  changes,  as  well  as  issued  queries  to  operators  for 
mission  status  updates. 

In  the  interface  depicted  in  Fig.  3,  operators’  primary 
tasks  are  to  identify  targets  found  by  the  network  of  UxVs 
and  approve  weapon  launches.  Once  a  target  is  found, 
the  operator  is  alerted  to  perform  a  target  designation  task 
(i.e.,  hostile,  unknown,  or  friendly),  along  with  assigning 
an  associated  priority  level  (i.e.,  high,  medium,  low).  In 
addition,  while  the  UxVs  are  capable  of  determining  and 
negotiating  their  own  search  patterns,  if  operators  are 
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Fig.  3.  Map  display . 


unhappy  with  the  UxV-de  ter  mined  search  patterns,  they 
can  create  new  search  tasks.  This  occurs  by  operators  right- 
clicking  on  those  areas  they  feel  need  more  attention, 
which  brings  up  a  menu  that  allows  the  operators  to 
prioritize  the  tasks  and  give  windows  of  desired  coverage. 
This  insertion  of  a  search  task  by  the  operator  in  effect 
forces  the  decentralized  algorithms  to  reallocate  the  UxVs 
such  that  the  operators’  desired  tasks  will  be  added  to  the 
task  list.  This  human-automation  interaction  scheme  is 
one  of  high  level  goal-based  control,  as  opposed  to  more 
low-level  vehicle-based  control.  Unlike  present-day  opera¬ 
tions  where  multiple  operators  command  a  single  vehicle, 
for  one  operator  to  effectively  control  multiple  vehicles, 
the  operator  must  control  the  network  via  high-level  tasks 
as  opposed  to  individual  vehicle  commands. 


A  performance  plot  in  the  lower  left  corner  of  Fig.  3 
gives  operators  insight  into  the  automated  planner  per¬ 
formance  as  the  graph  shows  expected  performance  (given 
an  a  priori  cost  function)  compared  to  actual  performance. 
When  the  automation  generates  a  new  plan  with  a  pre¬ 
dicted  performance  that  is  10%  higher  than  the  current 
plan’s  performance,  the  replan  button  in  Fig.  3  turns  green 
and  flashes.  This  illuminated  replan  button  indicates  that  a 
new  plan  is  ready  for  operator  approval,  and  when  the 
replan  button  is  selected,  the  operator  is  taken  to  the 
schedule  comparison  tool  (SCT,  Fig.  4). 

The  three  geometrical  forms  at  the  top  of  the  SCT  in 
Fig.  4  are  configural  displays  that  enable  the  operator  to 
quickly  compare  three  schedules:  the  current,  working, 
and  proposed  schedules.  Configural  displays  allow 


Fig.  4.  The  schedule  comparison  tool. 
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operators  the  ability  to  utilize  more  efficient  perceptual 
processes  rather  than  cognitively  demanding  processes 
that  rely  on  memory  and  inference  [21].  This  reliance  on 
perceptual  reasoning  is  known  as  direct  perception,  and 
for  this  decision  support  display  in  Fig.  4,  the  goal  is  to 
graphically  depict  to  the  operator  how  much  area  will  be 
searched  given  the  plan,  as  well  as  how  many  tasks  of  the 
differing  priorities  will  be  accomplished. 

In  Fig.  4,  the  left  form  (gray)  is  the  current  UxV 
schedule.  The  right  form  (green)  is  the  latest  automation 
proposed  schedule.  The  middle  schedule  (blue)  is  the 
working  schedule  that  results  from  the  user  modifying  the 
plan  by  querying  the  automation  to  assign  particular  tasks. 
The  rectangular  grid  on  the  upper  half  of  each  shape  re¬ 
presents  the  estimated  area  that  the  UxVs  will  search  ac¬ 
cording  to  the  plan.  The  hierarchical  priority  ladders  show 
the  percentage  of  tasks  assigned  in  high  (top),  medium 
(middle),  and  low  (bottom)  priority  levels.  Thus,  in  keep¬ 
ing  with  the  direct-perception  paradigm,  the  shape  that  is 
filled  with  the  most  color  depicts  the  best  plan.  In  addition, 
while  the  SCT  button  turns  green  whenever  the  automa¬ 
tion  determines  a  10%  better  plan,  operators  can  select  this 
button  at  anytime  to  rearrange  the  task  list  if  desired. 

When  the  operator  first  enters  the  SCT,  the  working 
schedule  is  identical  to  the  proposed  schedule.  The  opera¬ 
tor  can  conduct  a  “what  if”  query  process  by  dragging  the 
desired  unassigned  tasks  into  the  large  center  triangle  in 
Fig.  4  labeled  “assign.”  This  query  forces  the  automation  to 
generate  a  new  plan  if  possible,  which  then  becomes  the 
working  schedule.  The  configural  display  of  the  working 
schedule  alters  to  reflect  the  changes  of  newly  assigned 
tasks.  However,  due  to  possible  resource  shortages,  all  tasks 
may  not  be  assigned  to  the  UxVs,  which  is  representative  of 
real-world  constraints.  When  a  task  cannot  be  assigned,  the 
task  that  was  dragged  into  the  center  triangle  pops  out, 
representing  the  case  where  the  automated  planners  could 
not  honor  the  operator’s  requests  due  to  a  constraint  viola¬ 
tion  in  time,  fuel  required,  or  priority  conflict  (i.e.,  only 
tasks  with  higher  priority  could  be  accommodated). 

The  working  schedule  configural  display  updates  with 
every  individual  query  so  that  the  operator  can  quickly 
compare  the  three  schedules  based  on  quantity  of  color. 
This  querying  process  represents  a  more  collaborative 
effort  between  the  human  and  automation,  which  has  been 
shown  to  improve  operator  performance  and  situation 
awareness  in  similar  complex  settings  [22].  Ultimately,  the 
operator  either  accepts  the  working  or  proposed  schedule 
or  can  cancel  to  continue  with  the  current  schedule. 

The  system  is  designed  so  that  either  both  screens  can 
be  displayed  if  the  operator’s  ground  control  stations  sup¬ 
port  such  screen  real  estate,  or  the  screens  alternate  if  only 
a  single  display  is  available.  The  single  display  paradigm 
was  used  in  the  experiment  detailed  in  the  next  section 
since  the  expected  environment  was  one  of  an  expedi¬ 
tionary  controller,  i.e.,  a  human  moving  through  the  envi¬ 
ronment  with  only  a  laptop  to  control  the  UxVs. 


III.  THE  EXPERIMENT 

In  order  to  determine  the  impact  of  human  interaction 
with  the  decentralized  vehicle  network  on  mission  per¬ 
formance,  a  human-on-the-loop  experiment  was  con¬ 
ducted  and  the  results  compared  to  those  of  a  perfectly 
obedient  human  who  always  agreed  with  the  automated 
planner’s  proposed  schedule.  For  this  experiment,  parti¬ 
cipants  were  responsible  for  one  rotary-wing  UAV,  one 
fixed  wing  UAV,  one  unmanned  surface  vehicle  (USV), 
and  a  WUAV.  The  UAVs  and  USVs  were  responsible  for 
searching  for  targets,  which  each  had  a  unique  ID  internal 
to  the  system  so  ground  truth  was  always  known  to  the 
experimenters  for  later  analysis.  The  targets  continually 
moved  on  predetermined  paths  (unknown  to  the  UxVs  and 
operators),  so  there  was  no  guarantee  that  the  targets 
would  be  found.  Once  designated,  hostile  targets  were 
tracked  by  one  or  more  UxVs  until  the  human  operator 
approved  WUAV  missile  launches.  For  the  purposes  of  this 
experiment,  it  was  assumed  that  the  UxVs  could  not  be 
destroyed. 

The  experiment  was  conducted  on  a  Dell  Optiplex 
GX280  with  a  Pentium  4  processor  and  an  Appian 
Jeronimo  Pro  4-Port  graphics  card.  The  display’s  resolution 
was  1280  x  1024  pixels.  To  familiarize  each  subject  with 
the  interfaces  in  Figs.  1  and  2,  a  self-paced,  slide-based 
tutorial  was  provided,  as  well  as  a  practice  session.  The 
training  took  approximately  30  min  to  complete.  Practice 
was  followed  by  three  10-min  test  sessions,  representing 
each  of  the  three  possible  replan  intervals,  discussed  in 
detail  in  the  next  session,  in  a  counterbalanced  and  ran¬ 
domized  order.  Each  scenario  was  different,  but  similar  in 
difficulty.  The  interface  recorded  all  operator  actions. 

Because  it  has  been  previously  established  that  the  rate 
at  which  an  automated  scheduling  tool  prompts  the  user 
for  intervention  can  negatively  influence  UAV  operator 
performance  due  to  high  workload  [22],  the  subjects  in  this 
experiment  were  all  exposed  to  three  different  replanning 
intervals  of  30,  45,  and  120  s.  This  means  that  the  operator 
was  prompted  by  the  green  illumination  of  the  replan 
button  and  an  aural  replan  alert  every  30/45/120  s,  indi¬ 
cating  an  improved  schedule  was  available.  In  actual  use, 
such  automated  planners  would  not  likely  generate  plans 
with  such  regularity.  However,  the  rate  of  operator 
prompting  was  fixed  at  these  intervals  (with  a  standard 
deviation  of  ±5  s)  for  experimental  control.  OPS-USERS 
can  typically  generate  a  new,  theoretically  improved  plan 
once  per  every  10  s,  so  these  new  plans  were  simply  sup¬ 
pressed  until  the  correct  window  for  that  session  replan 
interval  was  achieved. 

The  original  subject  population  consisted  of  31  sub¬ 
jects.  However,  analysis  of  the  resulting  data  through 
statistical  clustering  revealed  that  only  approximately  one- 
third  of  the  subjects  (N  =  9)  actually  followed  instructions 
to  collaborate  with  the  automation  and  evaluate  the  auto¬ 
mation’s  plans  at  the  30/45/120  s  intervals.  These  subjects 
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were  labeled  consenters,  in  that  they  consented  to  the 
replanning  schedule,  although  they  did  not  necessarily 
agree  to  the  proposed  plan,  i.e.,  the  consenters  changed 
the  automation’s  proposed  schedule  about  one-third  of  the 
time.  Consenters  had  statistically  significantly  improved 
performance  over  the  remaining  22  subjects  who  were 
labeled  dissenters  (who  essentially  ignored  the  automa¬ 
tion)  or  mixed  consenters  (who  only  sometimes  agreed 
with  the  automation)  in  terms  of  when  to  replan. 

Characteristics  of  the  consenters  were  higher  than 
average  video  gaming,  as  well  as  no  military  experience, 
i.e.,  those  participants  with  military  experience  had  nega¬ 
tive  attitudes  towards  UVs.  An  in-depth  analysis  of  the 
consenter  versus  dissenter  performance  is  detailed  else¬ 
where  [23],  but  for  the  purposes  of  this  paper  which 
assesses  the  added  value  of  human-automation  collabora¬ 
tion,  only  the  data  from  the  consenter  group  will  be  con¬ 
sidered  since  they  were  statistically  superior  performers  as 
compared  to  the  remaining  22  participants.  For  the  re¬ 
mainder  of  this  paper,  this  consenting  group  will  be  called 
collaborators  since  they  generally  agreed  to  work  with  the 
automation,  with  occasional  input  to  redirect  the  auto¬ 
mation  as  necessary.  They  did  not  necessarily  agree  with 
the  automation,  on  average  they  disagreed  with  30%  of 
automation-generated  plans  and  conducted  “what  if” 
queries  to  generate  their  own  slightly  modified  plans. 

After  the  human-on-the-loop  experiment,  the  second 
portion  of  this  experiment  was  conducted  during  which  a 
single  operator  played  the  role  of  a  perfectly  obedient 
operator.  This  operator  completed  a  total  of  27  counter¬ 
balanced  trials  (nine  each  at  the  30/45/120  s  intervals). 
This  operator  always  agreed  with  the  replanning  rates,  i.e., 
always  entered  the  SCT  when  prompted,  and  also  agreed 
with  the  proposed  schedule,  i.e.,  the  operator  never  at¬ 
tempted  to  change  the  plan.  In  addition,  this  operator 
never  inserted  new  unprompted  search  tasks,  in  effect 
always  trusting  that  the  automation  was  performing  the 
optimal  search  and  tracking  task  allocation.  The  use  of  a 
single  operator  reduced  variability  in  response  time,  which 
was  generally  less  than  500  ms,  typical  for  such  point-and- 
click  behavior  that  requires  no  decision  making  on  the  part 
of  the  operator. 

Performance-dependent  variables  for  both  phases  of 
the  experiment  included  the  percentage  of  area  covered 
(which  is  critical  since  one  primary  task  of  the  UxVs  was  to 
cover  as  much  area  as  possible),  number  of  targets  found, 
number  of  hostile  targets  neutralized,  and  the  average 
operator  utilization  for  a  given  session.  Utilization,  a 
workload  measure,  is  the  percent  time  an  operator  is  busy 
over  the  entire  mission.  Operators  were  considered  busy 
when  performing  one  or  more  of  the  following  tasks: 
creating  search  tasks,  identifying  and  designating  targets, 
approving  weapons  launches,  interacting  via  the  chat  box, 
and  replanning  in  the  SCT.  Monitoring  time  was  not  in¬ 
cluded  in  percent  busy  time.  Previous  research  has  shown 
that  in  single-operator  goal-based  control  of  multiple 


entities,  operator  performance  can  significantly  drop  when 
tasked  greater  than  70%  in  terms  of  utilization  [24],  [25]. 

IV.  RESULTS  AND  DISCUSSION 

Since  this  mission  was  one  of  searching,  tracking  found 
targets,  and  neutralizing  hostile  targets,  each  of  these 
mission  elements  constitutes  an  effects-based  performance 
metric.  These  are  now  discussed  in  turn.  For  all  statistical 
tests  reported,  alpha  =  0.05. 

The  first  mission  performance  metric,  percentage  of 
area  searched  (Fig.  5),  illustrates  the  effectiveness  of  the 
overall  system  when  operators  collaborated  with  the  algo¬ 
rithm  (which  meant  changing  the  algorithm’s  plan  ap¬ 
proximately  30%  of  the  time),  as  compared  to  when  the 
human  was  perfectly  obedient  and  always  executed  the 
automation’s  plan.  The  overall  mixed  3x2  ANOVA  (with 
repeated  measures  across  the  three  replanning  intervals) 
shows  a  statistically  significant  difference  between  collab¬ 
orative  and  obedient  humans  [F(l,  24)  =  13.3,  p  <  0.001]. 
There  was  no  overall  main  effect  for  the  replanning  inter¬ 
val  [F(2,24)  =  1.7,  p  =  0.171]  or  interaction  between  re¬ 
planning  interval  and  obedient  versus  collaborative  human 
[F(2,  24)  =  0.711,  p  =  0.501].  Bonferroni  pairwise  com¬ 
parisons  reveal  that  for  the  30-  and  45-s  conditions,  the 
collaborative  human-automation  team  is  superior  to  the 
obedient  human  (p  <  0.001  for  both  comparisons),  but 
that  there  was  no  statistical  difference  at  the  120-s  interval 
(p  =  0.411).  This  indicates  that  for  the  faster  replan  in¬ 
tervals,  the  collaborative  human  provided  significant  ad¬ 
vantage  to  the  algorithm,  increasing  the  overall  area 
covered  by  up  to  30%,  on  average.  However,  this  large  gap 
closed  in  the  longer  replan  intervals,  with  the  collaborative 
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human  not  appreciably  improving  the  percent  area  covered 
compared  to  what  the  automation  essentially  planned  on 
its  own. 

It  should  also  be  noted  that  the  collaborative  human 
effort  for  searching  in  terms  of  area  covered  was  statis¬ 
tically  no  different  between  the  45-  and  120-s  replanning 
intervals  ( p  =  0.355),  but  significantly  lower  at  the  30-s 
interval  (p  =  0.002,  as  compared  to  120  s).  However,  the 
automation’s  performance  with  respect  to  search  linearly 
increased  as  replanning  interval  increased.  The  reason  for 
this  improvement  is  likely  due  to  the  fact  that  the  UxVs 
spend  a  greater  amount  of  time  searching  when  the  re¬ 
planning  interval  is  longer.  Search  is  a  spare  time  strategy, 
and  so  if  the  replan  interval  is  longer,  other  tasks  such  as 
track  or  neutralize  may  be  delayed  because  they  are  not 
released  to  the  distributed  tactical  planner  until  the  next 
replan.  These  results  suggest  that  there  was  a  plateau  for 
human  effort,  but  due  to  the  limited  experimental  condi¬ 
tions,  it  is  not  evident  in  the  data  that  the  automation 
reached  a  plateau.  This  will  be  discussed  further  in  a  sub¬ 
sequent  section. 

Related  to  the  area  searched  metric  is  the  targets  found 
metric,  since  simply  searching  the  most  space  is  not  a 
comprehensive  performance  metric,  i.e.,  it  represents  a 
subset  of  objectives.  Since  ultimately  the  goal  was  to  des¬ 
troy  as  many  targets  as  possible,  the  number  of  targets 
found  can  be  a  proxy  measure  for  the  quality  of  search. 
Using  the  same  3x2  ANOVA  model  discussed  previously, 
the  metric  of  percentage  of  available  targets  found  by  par¬ 
ticipants  (Fig.  6)  demonstrates  that  when  the  humans 
worked  with  the  planner,  significantly  more  targets  were 
found  across  all  replanning  intervals  (F(l,  24)  =  17.8,  p  < 


0.001),  than  when  the  human  perfectly  obeyed  automation 
directives.  Again,  there  were  no  other  main  effects  for 
replanning  rate  [F(2,  24)  =  0.470,  p  =  0.628]  or  interac¬ 
tion  [F(2, 24)  =  1.85,  p  =  0.179]. 

With  humans  occasionally  redirecting  the  automation 
(i.e.,  changing  the  automation’s  proposed  schedule  30%  of 
the  time),  up  to  50%  improvement  was  seen  in  the  number 
of  targets  found,  as  compared  to  if  the  automation  had 
been  left  alone.  Unlike  the  area-searched  metric,  there  was 
no  convergence  of  human-automation  performance  in 
that  the  human-automation  team  always  performed  more 
than  20%  better  for  every  replanning  rate  condition.  Once 
again,  the  collaborative  human  performance  appeared  to 
plateau  for  the  45-  and  120-s  intervals  in  that  they  were 
statistically  no  different  (p  =  0.891),  while  the  30-s  condi¬ 
tion  was  again  statistically  different  from  the  120-s  condi¬ 
tion  (p  =  0.046). 

Another  important  mission  performance  metric  was 
the  percentage  of  time  targets  were  successfully  tracked, 
i.e.,  not  lost.  Recall  that  targets  were  not  continually 
tracked,  in  that  UxVs  could  multitask,  and  attempt  to  track 
multiple  targets.  In  this  experiment,  we  were  not  able  to 
detect  any  measurable  human  operator  input  that  im¬ 
proved  upon  the  automation,  which  was  able  to  effectively 
track  targets  79%-99%  of  the  time.  This  is  not  to  say  that 
the  distributed  planner  would  perform  this  well  in  all 
cases,  but  just  for  the  numbers  of  targets  and  UxVs  in  this 
short  experimental  scenario,  the  planner  performed  very 
well  and  did  not  significantly  lose  target  tracks.  Moreover, 
since  human  operators  could  only  assist  help  with  finding 
lost  targets,  they  could  only  indirectly  influence  tracking 
targets.  However,  how  often  operators  inserted  and  reprio¬ 
ritized  search  tasks  could  affect  tracking  performance,  and 
we  leave  this  as  an  area  of  future  research. 

For  the  last  mission  performance  metric,  hostiles  neu¬ 
tralized,  there  was  no  statistical  difference  between  the 
collaborative  and  obedient  humans,  meaning  that  active 
human  management  of  this  process  did  not  improve  upon 
the  automation’s  performance  [F(l,  24)  =  0.001,  p  = 
0.974].  In  addition,  there  was  no  statistical  difference  in 
the  replanning  intervals  [F(2,  24)  =  1.36,  p  =  0.272],  so 
that  regardless  of  the  30-,  45-,  or  120-s  interval,  the  hos¬ 
tiles  neutralized  performance  was  consistent.  There  was 
no  significant  interaction  [F(2,24)  =  0.400,  p  =  0.675]. 
These  results  are  not  surprising,  given  that  of  the  three 
mission  performance  requirements,  human  intervention 
for  hostile  neutralization  was  minimal,  i.e.,  once  a  target 
was  designated  as  hostile,  execution  of  the  neutralization 
task  was  very  rule  based,  in  that  as  soon  as  a  target  was 
marked  hostile,  it  was  shadowed  by  a  WUAV  until  an  order 
was  received  to  neutralize.  Given  the  clearly  established 
rule  set  to  manage  this  process,  the  automation  was  very 
capable  of  positioning  the  WUAV  correctly,  without  hu¬ 
man  intervention. 

Given  that  workload  is  a  significant  concern  in 
single-operator  control  of  multiple  UxVs,  operator 
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utilization  (i.e.,  percent  busy  time)  was  measured  across 
the  three  replanning  intervals.  The  results  showed  no  sig¬ 
nificant  difference  in  utilization  across  the  three  replan¬ 
ning  levels  [F(2,  24)  =  2.817,  p  =  0.080],  which  is  not 
surprising  since  the  operators  in  this  analysis  represent  the 
top  performers.  The  average  percent  utilization  was  44% 
(±7%),  indicating  that  operators  were  not  overloaded  and 
well  below  the  threshold  of  70%. 

A.  Function  Allocation  Analysis 

The  results  from  the  experiment  clearly  show  that 
ultimately  mission  success  was  dependent  on  the  collab¬ 
oration  between  the  operators  and  the  automation,  and 
that  mutually  exclusive  function  allocation  either  within  or 
across  tasks  is  not  an  effective  solution.  Given  the  mission 
for  this  team  of  heterogeneous  UVs  of  search,  track,  and 
neutralize  enemy  targets  and  the  results  seen  from  this 
experiment,  Table  1  summarizes  the  function  allocation 
between  the  human  and  the  automation  for  the  various 
mission  components  that  promoted  the  best  overall  system 
performance,  as  motivated  by  the  statistical  results  in  the 
previous  section.  As  Figs.  5  and  6  demonstrate,  the  best 
search  performance  was  obtained  when  the  human  guided 
the  automation,  as  opposed  to  leaving  the  search  task  to 
just  the  automation.  These  results  are  somewhat  expected, 
as  the  human  ability  to  conduct  relatively  effortless  effec¬ 
tive  visual  searches  as  compared  to  computers  has  long 
been  recognized  [26].  However,  these  results  demonstrate 
just  how  much  advantage  human  guidance  in  the  search 
task  can  provide.  In  addition,  these  results  suggest  that 
human  performance  could  plateau  as  a  function  of 
increased  replan  times,  but  the  results  are  conflicting  in 
terms  of  the  capabilities  of  the  distributed  planners. 

For  this  set  of  experimental  conditions,  human  per¬ 
formance  in  terms  of  the  search  task  appeared  to  peak  at 
the  45 -s  replanning  interval,  with  no  statistically  signifi¬ 
cant  improvement  when  replanning  at  the  120-s  interval. 
For  the  percent  area  covered,  the  automated  planner  did 
worse  than  when  augmented  with  human  guidance  at  the 
30-  and  45-s  intervals,  but  at  the  120-s  interval,  it  per¬ 
formed  the  same.  This  is  possible  evidence  that  there  may 
have  been  a  replanning  interval  that  would  have  promoted 
better  automated  planning  performance  in  the  search  task. 
This  is  an  area  of  current  active  research,  as  it  is  not  ob¬ 
vious  how  to  determine,  given  an  uncertain  environment 
with  emergent  targets,  how  often  a  distributed  network  of 


UVs  should  communicate  in  order  to  maximize  the  search 
effort. 

While  the  benefit  of  the  human  ability  to  aid  in  the 
search  task  appeared  to  become  on  par  with  that  of  the 
automation  as  the  time  intervals  between  planning  grew 
longer,  this  relationship  did  not  hold  true  for  the  targets 
found  metric.  In  this  case,  the  human-automation  team 
far  exceeded  the  automation’s  ability  to  find  targets.  While 
percent  of  area  searched  is  an  important  metric  given  the 
stated  mission,  number  of  targets  found  demonstrates  the 
effectiveness  or  quality  of  the  search.  Thus,  humans,  aided 
by  the  automation,  were  better  able  to  judge  not  just  where 
the  automation  should  be  searching  in  terms  of  areas  left 
uncovered,  but  also  when  areas  should  be  revisited  and 
when  to  change  a  course  of  action  when  a  current  plan 
appeared  to  be  suboptimal.  It  is  this  combination  of  recog¬ 
nizing  both  where  and  when  to  search  that  led  to  superior 
human- automation  teaming  in  the  target-finding  task. 

While  some  tasks  benefitted  from  a  human-automation 
collaborative  effort,  there  are  some  tasks  that  should  be 
primarily  executed  by  either  just  the  UxVs  or  humans.  For 
example,  given  the  severe  consequences  of  misidentifica- 
tion  of  a  target  and  the  lack  of  technical  progress  that  has 
been  made  in  automated  target  recognition,  target  identi¬ 
fication  remains  a  task  that  is  expected  to  remain  a  human 
endeavor  for  the  near  future.  However,  improved  sensor 
technology  may  be  able  to  augment  and  aid  the  human  in 
this  complex,  high-risk  task. 

Alternatively,  tracking  performance  did  not  improve 
with  human  assistance,  suggesting  that  this  task  was  best 
left  to  automated  planners.  However,  one  caveat  is  that  a 
large  range  of  possible  scenarios  was  not  examined  in  this 
experiment,  so  under  different  conditions,  results  could 
have  varied.  However,  the  tracking  task  is  one  of  complex, 
rapid  computation  in  that  several  variables  must  be  con¬ 
sidered  such  as  locations  and  likely  trajectories  of  targets, 
locations,  and  movement  tracks  of  UxVs,  and  the  need  to 
revisit  targets  to  maximize  tracking  coverage  in  a  limited 
resource  environment.  Given  the  inability  of  a  single  ope¬ 
rator  to  be  able  to  make  such  rapid,  precise  computations 
in  a  dynamic  environment,  it  is  expected  that  automation 
would  be  able  to  execute  this  task  assuming  that  the  target 
was  clearly  identified.  Recent  developments  in  automated 
target  tracking  have  been  realized  in  operational  settings 
such  that,  for  example,  a  designated  target  in  the  form  of  a 
white  truck  can  be  automatically  tracked  as  it  moves.  It 


Table  1  Agent(s)  That  Promoted  the  Best  Task  Performance  in  the  Human-on-the-Loop  Experiment 


Agent  (s) 

Automation  Fed 

Human  +  Automation 

Human  Led 

3  ask 

Search 

✓ 

Target  Identification 

✓ 

Target  Track 

4/ 

Neutralize  Hostiles 

✓ 
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remains  to  be  determined  how  humans  could  augment  or 
assist  in  these  real-world  tracking  tasks. 

Last,  it  should  be  noted  that  the  hostile  neutralization 
task  was  another  that  benefitted  from  a  collaborative 
human- automated  planner  effort.  This  task  is  a  colla¬ 
borative  one  because  the  human  operator  made  the  final 
decision  to  release  a  weapon,  which  is  a  stressful  and 
information-gathering  intensive  task.  The  automation  kept 
the  WUAV  in  place  so  that  if  and  when  a  command  for 
neutralization  was  received,  the  system  was  ready  to 
respond.  Thus,  the  operator  was  able  to  offload  lower  level 
position  keeping  tasks  to  the  automation,  while  reserving 
precious  cognitive  resources  for  an  overall  knowledge- 
based  task  of  firing  a  weapon  that  will  likely  have  grave 
consequences. 

V.  CONCLUSION 

In  the  future  concept  of  one  operator  supervising  multiple 
collaborative  UxVs,  the  potential  exists  for  high  operator 
workload  and  negative  performance  consequences.  As  a 
result,  significant  autonomy  is  needed  to  aid  the  operator 
in  this  multiple  UxV  control  task.  Due  to  the  dynamic  and 
uncertain  nature  of  the  environment,  control  of  collabo¬ 
rative  and  decentralized  UxVs  requires  rapid  automated 
replanning.  However,  as  demonstrated  in  this  study,  hu¬ 
man  management  of  the  automated  planners  is  critical,  as 
automated  planners  cannot  always  generate  accurate  solu¬ 
tions  for  every  combination  of  events.  Though  fast  and  able 
to  handle  complex  computation  far  better  than  humans, 
computer  optimization  algorithms  are  often  “brittle”  in  that 
they  can  only  take  into  account  those  quantifiable  variables 
identified  as  critical  during  the  design  stage  [27],  [28]. 

This  research  has  shown,  for  an  admittedly  narrow  set 
of  conditions,  that  given  a  decentralized  UV  network,  hu¬ 
man  guidance  can  provide  substantial  benefit  in  search 
tasks,  with  arguable  little  improvement  in  tracking  tasks. 
Most  importantly,  this  research  has  demonstrated  that 
there  is  a  shared  space  in  such  missions  for  human- 
automation  collaboration.  While  there  may  be  some  mis¬ 
sion  tasks  that  can  be  mutually  assigned  to  either  a  human 
or  UxV  (due  primarily  to  inherent  limitations),  tasks  exist 
that  improve  substantially  when  humans  and  automation 
work  together.  Similar  results  have  been  achieved  in 
human-guided  optimization  of  heuristic  search  algorithms 


in  other  scheduling  domains  [29],  as  well  as  in  single¬ 
operator  supervisory  control  of  a  swarm  of  UAVs  for  target 
pursuit  [30].  Current  work  is  underway  to  identify  char¬ 
acteristics  of  scheduling  and  planning  problems  and 
algorithms  that  make  them  particularly  suitable  for 
human-algorithm  collaboration. 

This  research  also  raises  a  set  of  new  questions  that 
deserve  further  consideration.  While  algorithms  such  as 
those  embedded  in  OPS-USERS  can  typically  generate  new 
schedules  in  just  seconds,  often  these  schedules  are  not 
optimal,  i.e.,  solutions  generated  by  the  predetermined 
static  cost  function  may  not  reflect  the  true  state  of  all 
variables  in  a  dynamic  command  and  control  scenario. 
Such  planners  will  consider  even  a  solution  that  is  1% 
better  than  a  previous  solution  as  a  “better  solution,”  but 
often  such  algorithms  cannot  account  for  “satisficing” 
behavior  [31]  that  ultimately  is  just  as  “optimal”  (from  the 
viewpoint  of  stakeholders).  Reducing  the  gap  between 
planner  assumptions  and  human  expectations  is  an  impor¬ 
tant  area  of  future  research.  Moreover,  in  this  study, 
allowing  more  time  between  replans  improved  the  search 
performance  metrics,  so  future  work  is  needed  to  objec¬ 
tively  determine  effective  replanning  intervals.  These  in¬ 
tervals  are  likely  dynamic  parameters,  and  more  research 
is  needed  to  determine  if  some  Pareto  front  exists  that 
could  predict  a  region  of  optimized  performance. 

However,  changing  the  replan  interval  to  better  suit 
planner  performance  could  introduce  new,  unintended 
consequences.  It  remains  an  open  area  of  research  to  de¬ 
termine  how  to  design  such  a  system  that  achieves  a  ba¬ 
lance  between  human  guidance  and  human  interference. 
Previous  research  has  shown  that  those  operators  that 
refuse  to  collaborate  with  the  automation  can  actually  de¬ 
grade  overall  system  performance  and  also  significantly 
increase  their  own  workload  [23].  Thus,  designing  such  a 
system  that  reduces  operator  workload  and  engenders  trust 
and  collaboration  is  critical.  ■ 
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